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ABSTRACT 



Noting that child advocates are increasingly implementing 
monitoring projects to track the status of former welfare families and are 
using the findings to inform their agendas, policymakers, and the general 
public, this issue brief introduces monitoring strategies and suggests issues 
for consideration in designing a monitoring project. Monitoring, defined as 
tracking a family's income, composition, and other social and economic 
indicators, is useful for illustrating trends and highlighting what is 
happening to children and families. The brief highlights issues that should 
be considered in developing a monitoring project: (1) envisioning the final 

product; (2) choosing a focus; (3) identifying and accessing the target 
population; (4) ensuring statistical validity; and (5) identifying resources. 
Monitoring methodologies discussed in the brief are surveys, longitudinal 
tracking of families, focus groups, and the use of administrative data. 
Examples of various methodologies are highlighted in descriptions of 
individual state efforts. Also considered are issues related to selecting 
appropriate measures and maintaining credibility. Sample indicators are 
defined and operationalized. The brief also discusses the importance of 
communicating the project's results, including identifying target audiences, 
developing a message, connecting with the media, and disseminating the 
information. The brief concludes by noting that child advocates must remind 
the public and policymakers that the success or failure of welfare reform 
should be judged on the basis of its impact on the well-being of children and 
families. (Contains 20 notes.) (KB) 
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By Theresa J, Feeley and Sheri A. Brady 1 



he passage oj the Personal Responsibility and 
Work Opportunities Reconciliation Act (PRWO- 



POLICYM AKERS AND THE 
MEDIA TOUT DECLINING 
WELFARE ROLLS AS EVIDENCE 
OF THE SUCCESS OF WELFARE 

reform. However, child 

ADVOCATES AND THE PUBLIC 
DEFINE SUCCESS NOT BY CASE- 
LOAD DECLINES BUT BY THE 
WELL-BEING OF THE FAMILIES 
AFFECTED BY WELFARE 

reform. Increasingly, 

ADVOCATES ARE IMPLEMENT- 
ING MONITORING PROJECTS TO 
TRACK THE STATUS OF THESE 

families. They use the 

RESULTS TO INFORM THEIR 
AGENDAS, POLICYMAKERS, AND 



n the basis of caseload 
declines, politicians and 
the media have declared 
welfare reform a success. 
However, advocates and the 
public consider a more accu- 
rate measure of the success of 
welfare reform to be the well- 
being of families moving from 
welfare to work . 4 Many fami- 
lies that have left the welfare 
rolls are hardly self-sufficient; 
they have little or no earned 
income or need support serv- 
ices which they do not receive. 
The concern that welfare 
reform has left holes in the 
social safety net has prompt- 
ed some child advocacy 
organizations to monitor the 
status of families affected by 
welfare reform. 
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THE GENERAL PUBLIC. THIS 
ISSUE BRIEF INTRODUCES 
MONITORING STRATEGIES 




AND SUGGESTS ISSUES FOR 
ON SI DERATION IN DESIGNING 
A MONITORING PROJECT. 




Monitoring, in this instance, 
refers to the tracking of a fam- 
ily’s income, composition, and 
other social and economic 
indicators. Monitoring can 
describe the status of family 
and children affected by a pro- 
gram or policy, but it cannot 
prove a causal relationship 
between specific policies and 
the indicators. That is, moni- 
toring can demonstrate that 
people are leaving the welfare 
rolls, but it cannot determine 
whether welfare reform, the 
economy, or some other fac- 
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RA) in August 1996 transformed this nation’s family 
cash assistance program into a program emphasizing 
temporary assistance for families transitioning into the 
workforce. Since then, welfare rolls have declined by 
approximately 35% 2 , a fact attributed to a combination 
of a strong economy and changes in welfare policy. 
PRWORA imposed a 60-month lifetime limit on federal 
cash benefits and gave states the discretion to legislate 
even shorter time limits. As of January 1, 1999, families 
in ten states began to meet time limits. Time limits will 
begin to expire for families in an additional ten states 
this year. 3 Thus, the decline in welfare caseloads is 
expected to continue. 
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tor is the cause. Causality can only be determined through sci- 
entifically rigorous evaluations employing an experimental 
design . 5 Lack of resources and access to confidential informa- 
tion hinder the ability of nongovernmental organizations to 
conduct such evaluations. However, monitoring, while it can- 
not prove causation, is useful for illustrating trends and high- 
lighting what is happening to children and families. Child 
advocates are using the information from their monitoring proj- 
ects to identify families in need of additional support, pinpoint 
policies in need of reform, and educate policymakers and the 
general public. 

The purposes of monitoring vary, as do the means by which it 
is done. Thus, regardless of their size or resources, all child 
advocacy organizations should be able to identify and imple- 
ment a monitoring project that will enhance the work that they 
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are already doing on behalf of children. 
This issue brief is intended to assist child 
advocates in deciding whether and how 
to implement a monitoring project. It 
presents subjects for consideration and 
provides an overview of monitoring tech- 
niques and issues. Child advocacy organi- 
zations interested in pursuing monitoring 
further should contact the National 
Association of Child Advocates (NACA) 
for additional resources and. support. 



Considering a 
Monitoring Effort 



M onitoring can be an invaluable 
advocacy tool. However, it also 
has the potential to consume 
enormous amounts of staff time 
and monetary resources. Upfront 
planning will minimize the monitoring 
organization’s investment and maxi- 
mize the project’s value in the long run. 
This paper describes the issues for con- 
sideration step by step; in practice, the 
process may be less linear. 



Envisioning the Final Product - The single 
most important step to ensure the 
success of a monitoring project is decid- 
ing upon its purpose at the outset. The 
eventual use of the information should 
guide the project’s design. Some groups 
may wish to use the information inter- 
nally to inform their priorities. Others 
may wish to use the results to plan and 
support their legislative and policy 
agendas . 6 These disparate purposes 
should be reflected in the organization’s 
project design. 



A monitoring project’s purpose is particu- 
larly relevant when deciding the balance 
between qualitative and quantitative data 
to be gathered. For example, the anecdot- 
al information gathered during a series of 
focus groups with current and former 
welfare recipients or service providers 
may be sufficient to inform a child advo- 
cacy organization’s internal priorities. It 
can also inform child advocates about 
structural problems which they can then 
communicate to policy makers. On the 
other hand, organizations intending to 
use their monitoring results to educate 
policymakers about the status of families 



will need quantitative data to make the 
case that trends are widespread enough to 
merit a policy solution. Incorporating 
qualitative information such as testimoni- 
als and personal stories into the presenta- 
tion will enrich and explain the statistics 
and will likely attract more media atten- 
tion than quantitative data alone. 

Choosing a Focus - Once an organization 
has achieved consensus on the purpose of 
its monitoring effort, other issues for con- 
sideration will begin to crystallize. 
Prominent among these is the selection of 
a topic or focus for the monitoring proj- 
ect. Monitoring is most helpful when it is 
designed to answer specific questions that 
will help influence policy choices. For 
example, the project might review 
whether families leaving welfare are 
receiving other services and benefits for 
which they are eligible. Narrowing the 
focus early in the project ensures the 
development of a monitoring instrument 
which will elicit relevant information. 

Later, having a specific topic will expedite 
the interpretation and reporting of collect- 
ed data. The following list suggests several 
methods of topic development: 

■ Review the law — Since most organi- 
zations will be monitoring in 
response to a change in state or local 
policy, a review of the relevant law 
may prompt the choice of topic. 

■ Examine existing data — An analysis 
of existing data — whether adminis- 
trative or from other monitoring 
efforts — may reveal gaps and/or 
inconsistencies, thereby suggesting 
specific policies or populations in 
need of further monitoring. 

■ Tap internal resources — Staff and 

coalition partners are a tremendous 
resource. Lead them in a brain- 
storming session about topics that 
would benefit from monitoring. Ask 
them to reflect on conversations 
with policymakers, service providers, 
and low-income families as well as 
articles and editorials which they 
have read. From their research, they 
may also be familiar with issues that 
have arisen in other states and at the 
federal level. O 
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■ Consider what others have done - 
Advocacy organizations in other 
states or localities may have already 
identified and begun to monitor 
issues which are also relevant in 
your community. The National 
Association of Child Advocates has 
collected monitoring materials from 
around the country and can serve as 
a resource. 

■ Consult the experts — The people 
most familiar with the barriers to self- 
sufficiency or accessibility of services 
are current and former welfare recipi- 
ents and the service providers or legal 
services attorneys with whom they 
are in regular contact. Ask for then- 
suggestions and input via focus 
groups or one-on-one interviews. 

Identifying and Accessing the Target 
Population - The topics chosen will 
essentially dictate the target population, 
that is, the families whose status is 
being monitored. For example, if an 
organization chooses time limits as a 
topic, their target population will be 
families forced off of welfare at the expi- 
ration of their time. If an organization 
chooses work supports, their target 
population will be former recipients 
who left the system because of employ- 
ment or who have since gained employ- 
ment. Another potential target group for 
a monitoring effort is families diverted 
from welfare. Organizations should con- 
sider whether they want to collect infor- 
mation on a baseline group as well as 
their target population. 

The true challenge is not identifying the 
target population, but accessing it. 
Ideally, the monitoring organization 
would have access to state records or to 
state premises (at which they could do 
surveying); however, few child advocacy 
organizations have such collegial rela- 
tionships with state agencies. 
Additionally, state privacy laws may pre- 
vent the agencies from sharing the per- 
sonal information of current or former 
recipients. Because of these obstacles, 
child advocates may have to rely upon 
their relationships with independent 
service providers. 
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Listening to the Experts: Focus Groups in South Carolina 7 and Wisconsin 8 



m ™ ACA member the Alliance for South Carolina’s 

1 Children conducted 12 focus groups with stakehold- 
I ui ers we ^ are re f° rm i n order to understand the possi- 
I ^ ble implications of the state’s TANF plan. Four focus 
1 U groups were held in each of three counties. In order 
to get a complete picture of welfare reforms effects, these 
focus groups were held with different stakeholders. One 
consisted of Department of Social Service (DSS) staff; anoth- 
er of civic and faith leaders and staff from community agen- 
cies. The remaining two focus groups were comprised of 
low-income women who were denied enrollment in welfare, 
had their benefits reduced or eliminated, or left the rolls vol- 
untarily. Because of confidentiality concerns, DSS initially 
chose to recruit client participants rather than release client 
names to the Alliance for recruiting. However, a poor 
response forced the Alliance to recruit participants from the 
community. Each participant was paid $10 and offered 
assistance with transportation and child care. The focus 



groups were facilitated by Alliance staff who had been 
trained by a university-based expert. 

The Wisconsin Council on Children and Families (WCCF) 
also conducted focus groups with service providers and 
employers. The purpose of these groups was to identify 
issues, both regionally and statewide, that needed further 
attention from policy makers, but WCCF discovered several 
benefits. The focus groups provided an opportunity for 
WCCF staff to network with service providers. In addition, 
because service providers were able to connect with the advo- 
cacy community, they learned to act as advocates at the local 
level. The service providers also offered WCCF access to cur- 
rent and former welfare recipients willing to be interviewed 
for case studies. WCCF notes that though beneficial, the 
focus groups were time consuming to arrange. After the first 
two or three groups, subsequent groups of providers did not 
provide new information. 



In choosing service providers with 
whom to work, monitors should be 
aware of the effect of their selections on 
the validity of the results. Monitoring 
projects sometimes miss the most suc- 
cessful families because they have 
ceased to need or access the services 
where monitoring is taking place. 
Monitoring projects may miss those 
experiencing the most hardship 
because, by definition, they are not 
receiving the services that they need. 
Further complicating matters, these 
families often move frequently and lack 
access to a telephone. Monitoring proj- 
ects strive to accurately record the expe- 
riences of the target population. To do 
so, it may be necessary to tap into a 
variety of service providers ranging 
from those providing emergency servic- 
es (e.g., homeless shelters, food banks) 
to those serving the broader low- 
income population (e.g., health clinics, 
subsidized day care or Head Start pro- 
grams). Reliance upon service providers 
may incur additional biases which 
reflect providers’ funding, philosophies, 
or affiliations. It is impossible to avoid 
bias completely. The important thing is 
for child advocates to be clear about any 
inherent biases when analyzing and pre- 
senting results. 
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Data Capabilities - Though obviously 
vital to a monitoring project, the capaci- 
ty for processing and analyzing data 
should not define the project. Though 
the internal data capacities of child 
advocacy organizations vary greatly, 
other resources (such as those described 
below) can often be employed to achieve 
the desired end product. Once the child 
advocacy organization and its partners 
have committed to specific tabulation 
and analytical processes, these processes 
should be taken into consideration in 
the selection of measures. 

Ensuring Statistical Validity — Quantitative 
monitoring projects should strive for sta- 
tistical validity, that is, insuring that the 
results represent the target population as 
accurately as possible. Ideally, a monitor- 
ing effort would include 100% of the tar- 
get population. However, few child advo- 
cacy organizations have the capacity to 
gather information from each member of 
the target population and will, thus, rely 
on sampling. Because samples do not rep- 
resent the entire population, results from 
sampling should be interpreted as approx- 
imations which, by definition, contain 
some degree of error. 9 Child advocates 
engaged in monitoring efforts should 
strive to minimize their sampling error, 
thereby increasing statistical validity. 10 



The single best way to reduce sampling 
error is to increase sample size. Child 
advocates should consult statisticians to 
help them determine a sample size 
appropriate for their monitoring effort. 

In general, the larger the sample, the 
more valid the analysis. Child advocates 
will likely desire to analyze data for both 
the entire target population and specific 
subgroups. Thus, it is necessary to have 
an overall sample size sufficient to allow 
for acceptable sampling errors within 
subgroups. Since sampling error is large- 
ly determined by sample size, child 
advocates should decide early in the 
monitoring process, and discuss with a 
statistician, the largest sample error 
which they deem to be acceptable. 

Randomization is also necessary to 
insure statistical validity. The goal of ran- 
domization is to minimize the likelihood 
that extrinsic influences will shape the 
sampled population in ways that do not 
accurately reflect the target population 
as a whole. For example, because many 
families use up their food stamps or 
cash assistance within the first few 
weeks of each month, surveys being 
administered at emergency service 
providers at the end of the month may 
overestimate the daily needs of families. 
Thus, in practice, randomization 
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requires participation by every “X”th 
person within the relevant population. 

In the aforementioned food stamp 
example, bias could be minimized by 
having service providers sample every 
fifth person they serve over the course 
of a month. If monitors are unable to 
achieve randomization, they should 
include appropriate caveats in presenta- 
tions of their results. 

Identifying Resources — Collaboration can 
enhance the value and scope of a moni- 
toring project. Groups from different 
backgrounds can contribute knowledge, 
resources, and skills that exceed those of 
any single organization. Collaboration 
has the potential to benefit the monitor- 
ing project at each stage: linking with 
other organizations before research 
begins will bring useful ideas, resources, 
and connections; during research, collab- 
oration can provide access to a larger 
and more diverse population, or to a 
broader range of issues; after the 
research is completed, a diverse coalition 
can disseminate the information to reach 
the broadest possible audience as effec- 
tively as possible. 11 Given the potential 
benefits that can result from collabora- 



tion, it is important to consider a variety 
of partners early in the monitoring pro- 
ject’s planning. 

■ Foundations — Advocates can apply for 
small grants from foundations that usu- 
ally support research rather than advoca- 
cy to help them defray the costs of the 
monitoring project. 

■ Grassroots organizations — Grassroots 
organizations may have access to and 
established relationships with the target 
population. Due to their familiarity with 
the target population, grassroot organiza- 
tions may be able to contribute ideas for 
the project’s topic. They may also be will- 
ing to supply staff for data collection and 
tabulation. 

■ Colleges and universities — A local college or 
university or their associated cooperative 
extension service may be able to supply 
technical information and support that an 
advocacy organization does not have in- 
house, such as statisticians or social scien- 
tists who can advise about question develop- 
ment, sampling, randomization, and data 
tabulation. Where faculty or staff are state 
employees, colleges and universities may be 
able to access data that, because of privacy 



concerns, a child advocacy organization 
would not be privy to. Students may be will- 
ing to volunteer or intern with the monitor- 
ing project Advocates may also work with 
universities that are already conducting 
research to add an advocacy component to 
the research. For example, NACA member 
the Wisconsin Council on Children & 
Families was included in a university grant 
to do policy analysis and advocacy using the 
research results from a university study. 

■ National advocacy organizations — National 
organizations, such as NACA, are often 
familiar with or coordinating monitoring 
efforts in multiple states and localities. 
They may be able to provide local organi- 
zations with sample materials as well as 
technical assistance in project design. 

■ Service providers — Because they are famil- 
iar with low-income families, the barriers 
that prevent such families from attaining 
self-sufficiency, and service delivery and 
implementation issues, service providers 
are a tremendous resource for any moni- 
toring effort. They can contribute to the 
development of a topic and specific ques- 
tions. Additionally, once the project is 
underway, they may be willing to collect 
data on their premises. 



Tracking Families Over Time: The Wisconsin Council on Children and Families ' 2 



T he Wisconsin Council on Children and Families 

(WCCF), a NACA member, conducted an 18-month lon- 
gitudinal study of approximately 20 families in two 
counties. To find participants for the study, WCCF con- 
tacted agencies that serve families on public assistance 
including Head Start, county departments of human services, 
and community centers. In order to be eligible for participa- 
tion, the families had to be involved with TANF/Pay for 
Performance or its supportive services. 

Interviewers contacted potential participants and explained 
the project: its design, confidentiality policy, the commitment 
that participation would entail, and information regarding the 
payment of stipends. Stipends were used as an incentive for 
continuing participation; families were paid $25 for the initial 
interview and $ 10 for each subsequent interview, with the 
exception of the final interview for which they were paid $25. 
Families were also asked to provide contact names in the 
event that they relocated. If a prospect agreed to participate, 
then both interviewer and subject signed a Research 
Participation Contract outlining all of the above details. 



Researchers interviewed each family to gather background 
information and baseline data including demographics, 
employment, educational and welfare histories, income, 
expenses, and opinions. Follow up interviews were done 
quarterly, and families were asked questions about employ- 
ment experiences, changes in child care arrangements, hous- 
ing, income, and expenses. Families were interviewed by the 
same person throughout the length of the study. 

WCCF reports that, by the end of the 18 months, most partici- 
pants, especially those that reported greater difficulties in earlier 
interviews, were “lost” (disconnected telephones, no forwarding 
addresses). This indicates a greater likelihood of transient and 
marginal lifestyles among many leaving welfare programs. Staff 
concluded that the project, though worthwhile, consumed 
more resources than originally anticipated, especially given the 
challenge of tracking participants’ living arrangements. 
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Combining Quantitative and Qualitative Data: 

The Washington Welfare Reform Coalition 14 

T he Washington Welfare Reform Coalition, in which NACA member The 

Children s Alliance participates, combines quantitative and qualitative monitor- 
ing data to educate state policymakers. The Coalition has as two of its four goals 
collecting statistics and personal stories about the impact of welfare reform. In 
addition, the Coalition distributed fact sheets about welfare reform and recruit- 
ed over 400 people for participation in local low-income advocacy groups. 

Coalition members survey welfare recipients and other low-income people using a 
state-specific adaptation of the Coalition for Human Needs survey. 15 The survey 
asks about changes in benefit receipt, additional sources of support, whether basic 
needs are consistendy being met, current employment status, and barriers to 
employment The survey results are used to make policy recommendations, influ- 
ence legislators, and to educate the media to work for accurate facts. 

The advocacy value of the data is amplified when combined with the personal sto- 
ries of low-income people and families. Survey respondents are given the opportu- 
nity to fill out a Reality Check addressed to the Governor (a previous phase of 
the project had the checks going to state legislators). The check includes the indi- 
vidual’s name, address, monthly income, and family size. Space on the back of the 
check allows for the individual to detail their monthly income and expenditures 
and to write their personal story. The checks are sent to the Governor in the hope 
of reminding him that policies affect the lives of real people. The checks were also 
released publicly at a press conference in conjunction with the release of the 
Kellogg Foundation s National Poll on Welfare Reform and Health Care Reform. 
The monitoring results garnered substantial press coverage, much of it questioning 
the state s welfare reform plan for the first time. The press coverage and concurrent 
advocacy raised the awareness of the public and elected officials about potential 
negative effects of welfare reform. 



Regardless of whether child advocacy 
organizations elect to conduct their own 
monitoring effort, they may wish to serve 
as policy consultants for other organiza- 
tions engaged in monitoring. 

Choosing a Methodology 

election of a monitoring method- 
ology should depend upon the 
purpose and subject of the moni- 
toring project. A variety of 
approaches are available, but some 
are more suited to particular purposes 
than are others. There is no single best 
approach. This section introduces the 
various options and briefly describes 
their strengths and weaknesses. 

Surveys — Surveys which gather informa- 
tion at a particular point in time are per- 
haps the most common monitoring tech- 
nique. They are relatively inexpensive 
and, if care is taken to ensure their statis- 
tical validity, can yield rich qualitative 
and quantitative results. A well-devel- 
oped and administered survey can elicit 
substantial information through the use 
of both open- and closed-ended ques- 
tions. Surveys are particularly useful in 
assessing whether a family or sub-group 
of families is experiencing hardships. 

Surveys allow for substantial flexibility. 
They can be administered over the 
phone, through the mail, in person, or 
through a combination of these meth- 
ods. Organizations can choose between 
developing their own surveys, using 
those developed by others, or adapting 
national surveys to meet their local 
needs. 13 In addition to gathering informa- 
tion from current or former welfare 
recipients, surveys can be used to gather 
information from service providers about 
process changes and how they are affect- 
ing the services they provide and the 
clients they serve. 

The relatively few drawbacks to surveys 
can usually be overcome. Of primary 
concern is obtaining an acceptable 
response rate. Because of the possibility 
that the experiences of nonrespondents 
differ from that of respondents, high 
response rates are important to main- 
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taining the credibility of a monitoring 
effort. For example, Manpower 
Demonstration Research Corporation 
has an internal minimum standard 
requiring an 80 percent response rate. 
Child advocates should discuss with 
their consulting statisticians whether it 
is preferable to have a relatively small 
sample size and a high response rate or 
a large sample with a low response rate. 
Because they often miss families who 
are homeless, have moved, or lack a 
telephone, surveys administered via 
mail or phone tend to have a relatively 
low response rate. Thus, a decision to 
administer surveys in person may result 
in a higher response rate as well as a 
greater diversity of respondents. Service 
providers may be willing to administer 
surveys themselves or to let volunteers 
survey clients on their premises. 



Surveys rely on individuals’ understand- 
ing, knowledge, and recollections. 
Trained interviewers can improve the 
survey accuracy somewhat; however, 
because many people forget or confuse 
details and dates, surveys are not the 
most reliable means by which to gather 
very specific data such as employment 
and financial information. 

Longitudinal Tracking of Families — This 
approach involves picking a group of 
families to track at certain intervals over 
a specified period of time. The benefit of 
longitudinal studies is that they provide 
greater detail and depth than the one- 
time snapshot received from a survey. 
For example, they can demonstrate how 
the status of families leaving welfare for 
employment changes over the course of 
several years. This approach requires 
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the ongoing cooperation of representa- 
tive families willing to participate in what 
can sometimes be an invasive and lengthy 
process. The major drawback of a longi- 
tudinal approach is that some families 
will likely drop out or disappear as the 
study progresses. One way to minimize 
the dropout rate is to offer stipends for 
each round of participation. Because of 
the difficulties and cost of tracking fami- 
lies over time, most advocacy organiza- 
tions will only be able to track a small 
number of families, minimizing the sta- 
tistical validity of the information collect- 
ed. Nonetheless, longitudinal studies can 
provide qualitative information unlikely 
to be collected elsewhere. (See box on 
Wisconsin’s study.) 

Focus Groups - In this approach, the 
researcher brings together representatives 
from the target population to facilitate a 
discussion of relevant issues. Focus groups 
can be used alone or in conjunction with 
other monitoring tools. The size and 
nature of focus groups precludes quantita- 
tive analysis; however, because they elicit 
personal experiences and opinions they 
provide valuable insight to behavior and 
can enrich associated data. Focus groups 
are particularly useful for gathering infor- 
mation about service delivery and process 
issues, e.g., what happens when recipients 
apply for child care subsidies or what 
changes have taken place at service agen- 
cies since the passage of welfare reform. 
Focus groups elicit people’s views and 
experiences at a single point in time; how- 
ever, monitors may wish to consider host- 
ing repeated focus groups to learn how 
the experiences and opinions of the target 
population change over time. (See box on 
South Carolina and Wisconsin.) 



Administrative Data — Under the 1996 
welfare law, states are required to collect 
and report data that may be helpful to 
advocates who want to monitor the sta- 
tus of families affected by welfare 
reform. The federal government requires 
states to collect TANF-caseload data on 
families. In addition, states applying for 
a caseload reduction credit 16 or high per- 
formance bonus 17 are required to collect 
and submit additional data. Advocates 
can request and obtain from the state 



both the federally required data and any 
data collected by the state for its own 
purposes. For example, NACA member 
Arkansas Advocates for Children and 
Families (AACF) has obtained disaggre- 
gated data from the state and may con- 
duct its own analysis of the variables 
they want to monitor. 

While advocates may not be able to 
access state databases direcdy, they may 
wish to encourage states to use names or 
social security numbers to cross-match 
TANF data with other state data bases. 
For example, cross-matching with unem- 
ployment insurance data bases and the 
new hire registries used for child support 
enforcement will indicate whether former 
recipients are working, how much they 
are earning, what types of jobs they are 
getting, and whether they are advancing 
into higher paying jobs. Cross-matching 
TANF data with social service agency 
records will indicate which families are 
continuing to use other social services 
such as food stamps and Medicaid; cross- 
matching with child abuse registries can 
identify families in which abuse and neg- 
lect are occurring. While administrative 
data tend to be very accurate, analysis 
and cross-matching requires a high level 
of technical proficiency. Furthermore, 
because of privacy concerns, state agen- 
cies may be reluctant to release these 
data. Even if states are willing to share 
data, they will likely require a confiden- 
tiality agreement. Child advocacy organi- 
zations may be able to overcome these 
obstacles by partnering with state col- 
leges or universities. 

Selecting Appropriate 
Measures 18 

T he challenge in designing a moni- 
toring instrument is determining 
what information and data ele- 
ments are most relevant to the pro- 
ject’s purpose. Participants are 
often reluctant to fill out long or com- 
plex surveys, and excess data make data 
tabulation and analysis more cumber- 
some. This is where having narrowed the 
topic sufficiently will pay off. Extraneous 
information that could cloud or over- 
shadow the point can be discarded. 



For each of the topical areas chosen for 
the monitoring project, actual opera- 
tional measures need to be developed. 
Researchers do this by first breaking 
down the topic (e.g., self-sufficiency) into 
more specific concepts (e.g., labor mar- 
ket attachment) which can then be meas- 
ured in a variety of ways. For example, 
labor force attachment can be measured 
by counting when a client first gets a job 
and/or by the length of time he/ she 
remains in the job. These operational 
measures may vary dependent upon the 
target population and the unit of analysis 
(family, community, etc.) selected for 
monitoring. The chart on the following 
page lists sample concepts and associat- 
ed operational measures, some of which 
could potentially be broken down even 
further. The chart provides examples 
and is not intended to be exhaustive. 



Maintaining Credibility 

A dvocates must acknowledge the 
limitations of project design and 
methodology during all phases of 
their monitoring project. Whether 
during the analysis of data, the 
development of findings and recommen- 
dations, or the creation of written materi- 
als, it is crucial that the data not be mis- 
represented or the conclusions overstat- 
ed. In this way, credibility is enhanced. 



As stated previously, monitoring does 
not establish causality. Thus, advocates 
should avoid strong claims about cause 
and effect. For example, if the monitor- 
ing results suggest that average family 
income has declined, advocates should 
resist making assertions that “welfare 
changes have hurt families.” Rather, 
they should use the results to establish 
that families are faring worse than they 
were previously and state that the data 
collected is consistent with the conclu- 
sion that welfare changes are to blame. 19 



Communicating the Results 




hild advocacy organizations may 
engage in monitoring for a variety of 
reasons. For some, the sole purpose 
is to inform their organization’s 
internal agenda. However, the vast 
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TOPIC 


DEFINITION 


OPERATIONAL MEASURES 


Dependent* 


The use of public assistance 
programs to meet basic 
economic needs. 


caseload counts; reasons for exit; re-entry 
counts; grant amounts; duration of welfare 
“spell”; use of other public services 
(e.g., Medicaid, food stamps, SSI) 


Hardship 


Maintenance of a minimal 
standard of living. 


homelessness (shelter occupancy); residential 
stability/mobility (eviction rates, doubling up 
of families in single-family residence); food 
insecurity (use of food pantries & soup 
kitchens); access to medical care 


- Workforce Attachment 


Demonstrated efforts at obtaining or 
maintaining employment. 


job entry; type of work (paid, subsidized, 
community); hours of work; enrollment in train- 
ing;job retention/ turnover; job advancement 



Economic Well-being 



Income and financial status. 



earned or unearned income; job benefits; 
child support 



Utilization of Support Services Whether families need and are application rates; eligibility rates; utilization 

accessing supportive services (child rates; duration of use 
care, food stamps, Medicaid, CHIP). 



Community Involvement Communities’ support of welfare 

relorm. 



Economic Development 






Family Formation i. 






Accessibility of jobs that encourage 
self-sufficiency. 



Individuals’ ability to form and 
maintain stable family bonds. 



public attitudes toward welfare; business/ 
community partnerships; charitable sendees 

changes in low-wage labor market; job creation; 
creation of iiew types of support services; 
availabUity of transportation to work sites 

TTvaraage/divorce rates; cohabitation rates; addi- 
tional/post-program-entry births; out-of-wedlock 
births; out : of-home placements; kinship care 



majority of child advocacy organizations 
intend to use the results of their monitor- 
ing project to educate various constituen- 
cies about the lives of low-income per- 
sons affected by welfare reform and 
about policy changes with the potential 
to improve those lives. Effective education 
and advocacy around monitoring results 
requires that advocates devise a commu- 
nication strategy for identifying target 
audiences, developing a message, con- 
necting with the media, and disseminat- 
ing the information. 

Identifying Target Audiences — The choice 
of target audiences is dictated by the 
purpose of the monitoring project. In 
fact, some organizations may choose 
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their topics based on whom they are 
trying to influence. The target audience 
often includes elected and appointed 
officials as well as other opinion lead- 
ers, such as business leaders. Advocates 
might also target service providers, the 
religious community, and journalists. 

Developing a Message — Advocates should 
develop a clear and concise message of 
not more than ten words 20 which is based 
on the results of the monitoring project. 
The message should be consistent with 
the values and mission of the advocacy 
organization and link the research find- 
ings to their impact on children. For 
example, if the monitoring results show 
that families are moving frequently 



because they are unable to afford rent, 
advocates may wish to point out that the 
possible effect on children is disruption 
of school and social relationships. In 
addition, advocates should develop talk- 
ing points which highlight the findings 
and are expressed in language which the 
general public can understand. The talk- 
ing points will help keep the message 
consistent and should guide all media 
interactions. 

Messages sometimes fail to communicate 
their intent. However, advocates can mini- 
mize the possibility of miscommunica- 
tion by testing messages with one or 
more focus groups prior to their dissemi- 
nation. Obtaining feedback from focus 
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groups that resemble the intended target audiences can enable 
advocates to both refine their messages and to anticipate ques- 
tions prior to dissemination. 

Connecting With the Media - Reaching journalists is a critical com- 
ponent of educating policy makers and changing public opinion. 
The ideal time for advocates to learn about the media and to 
forge relationships with journalists is before they are needed. 
Gaining an appreciation for the culture and rhythm of the media 
industry and building relationships of trust and credibility will 
improve the likelihood of receiving appropriate, accurate, and 
well placed press coverage. In addition to the traditional press 
conference, advocates should host media briefings and meet with 
editorial boards. These meetings offer journalists the opportunity 
to gain an in-depth understanding of the project’s findings. 

Disseminating the Information - Long reports can boost credibility 
and be valuable sources for future reference. Nonetheless, as part 
of their communications strategy, advocates should also create 
summaries, fact sheets, and short papers with a minimum of 
data. These shorter pieces tend to be particularly effective for 
reaching the media and policymakers. 

Materials related to the monitoring project should be disseminat- 
ed to the targeted audiences identified in the original project plan. 
Advocates can improve dissemination by asking coalition partners 
to distribute written materials within their circles, offer summaries 
in newsletters, and to host briefings in their communities. 
Advocates should also consider distributing summaries in public 
places such as libraries and city halls or in places which families 
frequent such as child care centers, clinics and grocery stores. 

In addition to written materials, advocates should select and 
train spokespersons (individuals from the population being 
studied can be effective here), establish speakers’ bureaus to 
reach community groups such as Kiwanis and PTAs, and write 
op-ed pieces and letters to the editor to remind the public of the 
monitoring project’s findings over time. 



Conclusion 

In light of assertions equating caseload declines with the 
success of welfare reform, child advocates must remind the 
public and policymakers that the success or failure of wel- 
fare reform should be judged on the basis of its impact on 
the well-being of children and families. Though monitoring 
is not able to establish causality, it is an important tool for 
keeping the focus on whether families are faring better or 
worse in this age of welfare reform. Monitoring projects do 
not have to be complicated, cumbersome, or expensive to 
be effective, and child advocacy organizations interested in 
pursuing monitoring have tremendous flexibility in the 
design of their projects. NACA looks forward to serving as a 
resource for organizations hoping to hold their states and 
localities accountable through monitoring. 

9 
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